Degrees of Freedom of a Communication Channel: 
Using Generalised Singular Values 

Ram Somaraju and Jochen Trumpf 
Abstract 

A fundamental problem in any communication system is: given a communication channel between a transmitter 
and a receiver, how many "independent" signals can be exchanged between them? Arbitrary communication channels 
that can be described by linear compact channel operators mapping between normed spaces are examined in this 
paper. The (well-known) notions of degrees of freedom at level e and essential dimension of such channels are 
developed in this general setting. We argue that the degrees of freedom at level e and the essential dimension 
fundamentally limit the number of independent signals that can be exchanged between the transmitter and the 
receiver. We also generalise the concept of singular values of compact operators to be applicable to compact 
operators defined on arbitrary normed spaces which do not necessarily carry a Hilbert space structure. We show 
how these generalised singular values can be used to calculate the degrees of freedom at level e and the essential 
dimension of compact operators that describe communication channels. We describe physically realistic channels 
that require such general channel models. 
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I. Introduction 

The basic consideration in this paper can be stated as 
follows: given an arbitrary communication channel, is 
it possible to evaluate the number of independent sub- 
channels or modes available for communication. Though 
this question is not generally examined explicitly, it 
plays an important role in various information theoretic 
problems. 

A rigorous proof of Shannon's famous capacity re- 
sult [1] for continuous-time band-limited white Gaussian 
noise channels requires a calculation of the number 
of approximately time-limited and band-limited sub- 
channels (see e.g. [2, ch. 8] and [3,4]). This result can be 
generalised to dispersive/non-white Gaussian channels 
using the water-filling formula [1,2]. In order to use this 
formula, one needs to diagonalise the channel operator 
and allocate power to the different sub-channels or 
modes based on the singular values of the corresponding 
sub-channel. One therefore needs to calculate the modes 
and the power transferred (square of the singular values) 
on each one of these sub-channels to calculate the 
channel capacity. 

The water-filling formula has been used extensively 
in order to calculate the capacity of channels that use 
different forms of diversity. In particular, the capacity of 
multiple-input multiple-output (MIMO) antenna systems 
has been calculated using this water-filling formula for 
various conditions imposed on the transmitting and the 
receiving antennas (see e.g. [5] and references therein). 
Water-filling type formulas have been used for other 
multi-access schemes such as OFDM-MIMO [6] and 
CDMA [7] (see also Tulino [8, sec 1.2] and references 
therein). More recently, several papers have examined 
the number of degrees of freedornj available in spatial 
channels [9]— [13]. Questions of this nature have also 
been studied in other contexts such as optics [14] and 
spatial sampling of electromagnetic waves [15,16]. 

Both types of results, the modes of communication 
used for the water-filling formula and the number of 
degrees of freedom of spatial channels use the singular 

'Note that other terms such as modes of communication, essential 
dimension etc. have been used instead of degrees of freedom in some 
of these papers. 



value decomposition (SVD) theorem. One can use SVD 
to diagonalise the channel operator and the magnitude of 
the singular values determines the power transferred on 
each of the sub-channels. The magnitude of these singu- 
lar values can therefore be used to calculate the number 
of degrees of freedom of the channel (see e.g. [9,12]). 
However, the SVD theorem is only applicable to compact 
operators defined on Hilbert spaces. An implicit and 
valid assumption that is used in these papers is that 
the operators describing the communication channels are 
defined on Hilbert spaces. These results can therefore not 
be generalised directly to communication systems that 
are modeled by operators defined on normed spaces that 
do not admit an inner product structure. There are several 
instances of practical channels that can not be modeled 
using operators defined on inner-product spaces (see 
Section III- A I for examples). In this paper, we develop 
a general theory that enables one to evaluate the number 
of degrees of freedom of such systems. 

We wish to examine if it is possible to evaluate the 
number of parallel sub-channels available in general 
communication systems that can be described using 
linear compact operators. Any communication channel 
is subject to various physical constraints such as noise 
at the receiver or finite power available for transmission. 
If the channel can be modeled via a linear compact oper- 
ator, then these constraints ensure that only finitely many 
independent channels are available for communication. 
Roughly speaking, we call the number of such channels 
the number of degrees of freedom of the communication 
system (see Section JII] for a precise definition). Note 
that if the channel is modeled using a linear operator 
that is not compact then it will in fact have infinitely 
many parallel sub-channels, or some channels that can 
transfer an infinite amount of power (see Theorem 13.101 
below and the discussion following it). It could hence 
be argued that the theory presented in this paper is the 
most general theory needed to model physically realistic 
channels. 

We give novel definitions for the terms degrees of free- 
dom and essential dimension in the following section. 
Even though these terms have been used interchangeably 
in the literature, we distinguish between the two. The 
essential dimension of a channel is useful for channels 
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that have numbers of degrees of freedom that are es- 
sentially independent of the receiver noise level (e.g. 
the time-width/band-width limited channels in Slepian's 
work [17]). Also, we generalise the notion of singular 
values to compact operators defined on normed spaces 
and explain how these generalised singular values can 
be used to compute degrees of freedom and the essential 
dimension. 

A. Channel Model 

We assume that a communication channel between a 
transmitter and a receiver can be modeled as follows. 
Let X be a linear vector space of functions that the 
transmitter can generate and let Y be a linear vector 
space of functions that the receiver can measure. We 
assume the existence of a linear operator T : X — > Y 
that maps each signal generated by a transmitter to a 
signal that a receiver can measure. We also assume that 
there is a norm || • \\x on X and a norm || • ||y on Y. 
This model is very general and can be applied to various 
situations of practical relevance. 

For instance, consider a MIMO communication sys- 
tem wherein the transmitter symbol waveform shape on 
each antenna is a raised cosine. In this case we can 
think of the space of transmitter functions X to be (more 
precisely, to be parametrised by) the n-dimensional com- 
plex space C n that determines the phase and amplitude 
of the raised cosine waveform on each antenna. Here 
n is the number of transmitting antennas. Also, we 
can think of the space of receiver functions as C m , 
where m is the number of receiving antennas. T in this 
context is a channel matrix, representing the linearized 
channel operator that depends on the scatterers in the 
environment. 

Alternatively, consider a MIMO communication sys- 
tem in which the transmitter symbols are not fixed but 
can be any waveform of time. Suppose the symbol 
time is fixed to t s seconds. In this case, we can think 
of the space of transmitter functions, X, as the space 
£ 2 ([0, t s ], C n ) of C n -valued square integrable functions 
defined on [0, t s ]. Similarly, we can think of the space 
of receiver functions, Y, as the space £ 2 ([0, t s ], C m ). 
Again, T is the channel operator. 

Irrespective of the precise form of the underlying 
spaces X and Y, we always call elements of X transmit- 
ter functions and the elements of Y receiver functions. 
Also, we call the space X the space of transmitter 
functions and the space Y the space of receiver functions. 
In particular, we do not distinguish between the two 
different physical situations: a) the elements of X are 
functions of time and b) the elements of X are vectors 



in some finite dimensional space. This should cause no 
confusion and we use this convention for the remainder 
of this document. 

We now restrict ourselves to situations where there 
is a source constraint || ■ \\x < P that can be imposed 
on the space of transmitter functions X, and where the 
operator T is compact. Roughly speaking, the norm 
on the space of transmitter functions X captures the 
physical restriction that the transmitter functions can 
not be arbitrarily big, while the norm on the space of 
receiver functions can be interpreted as a measure of how 
big the received signals are compared to a pre-specified 
noise level. We therefore try to find how many linearly 
independent signals can be generated at the receiver that 
are big enough by transmitter functions that are not too 
big. The compactness of the operator T ensures that only 
finitely many independent signals can be received (see 
Section ITl-AI for examples of such channels). This vague 
idea is clarified further in the following two sections. 

B. Outline 

The remainder of this paper is organised as follows: in 
the next section we consider a finite dimensional exam- 
ple and motivate the definition of degrees of freedom. We 
also discuss several examples of practical communica- 
tion systems to which the theory developed in this paper 
may be applied. Section [III] presents the main results 
of this paper as well as formal definitions of degrees 
of freedom, essential dimension and generalised singular 
values. Conclusions are presented in Section[lV] Detailed 
proofs of the theorems in this paper are presented in the 
Appendix. 

Most of the material presented in this paper forms part 
of the first author's PhD thesis [18]. 

II. Motivation 

We motivate our definition of degrees of freedom 
at level e for compact operators on normed spaces by 
considering linear operators on finite dimensional spaces. 
Consider a communication channel that uses n transmit- 
ting antennas and m receiving antennas which can be 
mathematically modeled as follows. Let the current on 
the n transmitting antennas be given by x £ C n . This 
current on the transmitting antennas generates a current 
y € C m in the m receiving antennas according to the 
equation 

y = Hx. 

Here, H € c mxn j s the channel matrix. We can define 
the operator T : C n — > C m by x h-> y = Hx. Also, 
for n = 1,2,..., || • || = y/ (■)*(•), with (•)* denoting 
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the complex conjugate transpose, is the standard norm 
in C n . In this context, the norm determines the power of 
the signal on the antennas. 

The singular value decomposition theorem tells us 
that there exist sets of orthonormal basis vectors 
{vi, . . . , v n } C C n and {m, . . . , u m } C C' m such that 
the matrix representation for T in these bases is diagonal. 
Let be such a matrix with the basis vectors ordered 
such that the diagonal elements (i.e. the singular values 
of T) are in non-increasing order. A simple examination 
of the diagonal matrix proves that for all e > there exist 
a number N and a set of linearly independent vectors 
{yi, • • • , Yn} C C m such that for all x e 77,. ;■ (ofl 



inf 

ai,...,a^v 



H d x 



N 
i=l 



< e. 



For a given e, call the smallest number that satisfies the 
above condition Af(e). Note that the vectors yi, . . . , yjv 
span the space of all linear combinations of the left 
singular vectors of T whose corresponding singular 
values are greater than or equal to e. 

A simple examination of the diagonal matrix tells us 
that A/ r (e) is equal to the number of singular values of T 
that are greater than e and is hence clearly independent 
of the bases chosen. This leads us to our definition for 
degrees of freedom in finite dimensional spaces. 

Definition 2.1: Let T : C n — > C m be a linear operator 
and let e > be given. Then the number of degrees of 
freedom at level e for T is the smallest number N such 
that there exists a set of vectors yi , . . . , y^v £ C m such 
that for all x 6 i?i,C" (0) 



inf 

ai,...,a^v 



Tx 



A? 



=1 



a;y, 



< e. 



This definition is appropriate for the number of degrees 
of freedom because for a MIMO system the norm || • || 
represents the power in the signal. Suppose we wish 
to transmit N linearly independent signals from the 
transmitter to the receiver, and the total power available 
for transmission is bounded. Suppose further that the 
received signal is measured in the presence of noise. 
By requiring that x € -Bi,c™(0) we are constraining the 
power available for transmission. We model the noise 
by assuming that any two signals at the receiver can 
be distinguished if the power of the difference between 
the signals is greater than some level e. Similar ideas 
have been used for instance by Bucci et. al. [16] (see 
also [4,10,17]). According to this definition, the number 
of degrees of freedom is equal to the number of linearly 

2 Given a normed space X, r > and x £ X, B r ,x(x) denotes 
the closed ball of radius r centered at x 6 X . 



independent signals that the receiver can distinguish 
under the assumptions of a transmit power constraint 
and a receiver noise level represented by e. Note that 
we are making the implicit assumption that the power 
P is 1 in the above definition. This does not cause a 
problem because we can always scale the norm in order 
to consider situations where 1. 

The above definition was motivated using the singu- 
lar value decomposition theorem in finite dimensional 
spaces. It can therefore be easily generalised to infinite 
dimensional Hilbert spaces using the corresponding sin- 
gular value decomposition in infinite dimensional Hilbert 
spaces (see eg. [16,1 8 Jib- However, the singular value 
decomposition can only be used for operators defined on 
Hilbert spaces. It cannot be used for operators defined 
on general normed spaces. Observe that the definition for 
degrees of freedom above only depends on the norm || • || 
and not on the assumption that the underlying spaces C n 
and C m are Hilbert spaces. It will be shown in this paper 
that the above definition can be extended to compact 
operators defined on arbitrary normed spaces. 

Now consider the situation where the singular val- 
ues of the operator T show a step like behav- 
ior. For instance, suppose the singular values are 
{1, 0.9, 0.85, 0.5, 0.1, 0.05, .0005}. In this particular case 
the number of degrees of freedom at level e is equal 
to 4 for a big range of values of e and the number 
of degrees of freedom is essentially independent of the 
actual value of e chosen. Such a situation arises in several 
important cases (see eg. [4,9,14,16,17]). It would be 
useful to have a general way in which one can specify 
a number of degrees of freedom of a channel that is 
independent of the arbitrarily chosen level e. In this paper 
we provide a novel definition for such a number and call 
it the essential dimension of the channel. This definition 
is sufficiently general to be applicable to a variety of 
channels and quantifies the essential dimension of any 
channel that can be described using a compact operator. 

A. Examples 

As explained in section iLAl we assume that a commu- 
nication channel can be described using the triple X, Y 
and T. Here X is the space of transmitter functions, Y 
is the space of receiver functions and T is the channel 
operator and is assumed to be compact. As explained 
earlier in this section, if the spaces X and Y are Hilbert 
spaces and if the operator T is a linear compact operator 
then the well known theory of singular values of Hilbert 
space operators can be used to determine the number of 
degrees of freedom of such channels. However, if either 

3 Also compare with the time-bandwidth problem in [4,17], 
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one of the spaces X or Y is not an inner product space 
then one cannot use this theory. 

There are several practical channels that are best 
described using abstract spaces that do not admit an 
inner product structure. In this subsection, we consider 
three examples of such channels. In the first example, 
the measurement technique used in the receiver restricts 
the space of receiver functions. In the second one, the 
modulation technique used means that the constraints 
on the space of transmitter functions are best described 
using a norm that is not compatible with an inner 
product. The final example discusses a physical channel 
that naturally admits a norm on the space of transmitter 
functions that is described using a vector product and 
therefore does not admit an inner-product structure. 

Example 2.1: In any practical digital communication 
system, the receiver is designed to receive a finite set 
of transmitted signals. Suppose the transmitted signal is 
generated from a source alphabet {ii, . . . , tjv} and for 
simplicity assume that in a noiseless system each element 
from the source alphabet tj, 1 < i < N, generates a sig- 
nal r i , 1 < i < N, at the receiver. In the corresponding 
noisy system, the fundamental problem is to determine 
which element from the source alphabet was transmitted 
given the signal r = n + n was received. Here, n 
is the noise in the system. One common approach to 
solving this problem is to define some metric d(-, •) that 
measures the distance between two receiver signals and 
to calculate 

r = argmin d(r,ri). 

{ri,l<i<n} 

One concludes that the element from the source alphabet 
that corresponds to r' is (most likely) the transmitted sig- 
nal. Generally, this metric d(-, •) determines the abstract 
space Y of receiver function. 

Now consider a MIMO antenna system with n trans- 
mitting and m receiving antennas. Suppose that the re- 
ceiver measures the signals on the m receiving antennas 
for a period of r seconds. One can describe the received 
signal by a function y(t), where y : [0, r] — » C m . In 
order to implement the receiver one can use a matched 
filter if the shapes of all noiseless receiver signals are 
known. In this case the distance between two received 
signals can be described using the metric 

d(yi,y 2 ) = (T(yi(*) -ya(*))*(yi(*) -Y2 ' 

One can describe the space of receiver functions using 
the HUbert space £ 2 ([0, r],C m ) with the inner product 
defined by 

(yi,y 2 ) := / yt(t)y 2 (t)dt. 
Jo 



This is the common approach used in information theory. 

However, it is generally easier to measure just the 
amplitude of the received signal on each of the m 
antennas. In fact, in a rapidly changing environment it 
might not be possible to build an effective matched filter 
and therefore there is no benefit in measuring the square 
of the received signal. In this case the distance between 
any two signals can be described using the metric 

d(yi,y 2 ) = / \yi(t)-y 2 (t)\dt. 
Jo 

Here, one can describe the space of receiver functions 
using the Banach space >C 1 ([0, r], C m ) with the norm 
defined by 

llyll := / \y(t)\dt. 

Jo 

This channel therefore is best described using a normed 
space as opposed to an inner product space to model the 
set of receiver signals. 

Example 2.2: Consider a multi-carrier communica- 
tion system that uses some form of amplitude or angle 
modulation to transmit information. Suppose that there 
are n carriers and that the vector <f> = [<fii, . . . , <p n ] 
determines the modulating signal on each of the carriers. 
We can think of the modulating waveforms as the space 
of transmitter functions 

If amplitude modulation is used then the vector cj) 
determines the total power used for modulation. If the 
total power available for transmission is bounded then 
one might have an inequality of the form 

n 

5>! 2 <p. 

We can therefore describe the space of transmitter func- 
tions using the standard Euclidian space W % with inner 
product 

(xi,x 2 ) = xfx 2 . 

Now consider the case where angle modulation is 
used. In this case all the transmitted signals have the 
same power and the total power available for transmis- 
sion places no restrictions on the space of transmitter 
functions. However, the space of transmitter functions 
can be subjected to other forms of constraints. For 
instance, if frequency modulation is used then the max- 
imum frequency deviation used might be bounded by 
some number b to minimise co-channel interference (see 
e.g. [19, p. 110,513]). Similarly if phase modulation is 
used the maximum phase variation has to be less than 

4 In this case we do not consider the actual signal on the transmit- 
ting antenna (i.e. carrier + modulation) to be the transmitter function. 
Cf. the discussion in Subsection II- Al 
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±7r. This bound may also depend on other practical 
considerations such as linearity of the modulator. In 
this case one might constrain the space of transmitter 
functions as 



sup | 

Ki<n 



< b. 



The space of transmitter functions of this channel is best 
described using the n-dimensional Banach space 
with norm 



n 
oo 



||x|| = SUp |Xj|. 

l<i<n 

Example 2.3: In this final example we examine spatial 
waveform channels (SWCs) [18]. In SWCs we assume 
that a current flows in a volume in space and generates 
an electromagnetic field in a receiver volume that is 
measured [10,15,16,18]. Such channels have been used 
to model MIMO systems previously [10,12,13,15,16,18]. 
If a current flows in a volume in space that has a finite 
conductivity, power is lost from the transmitting volume 
in two forms. Firstly, power is lost as heat and secondly 
power is radiated as electromagnetic energy. So the total 
power lost can be described using the set of equations 



total 



lost 



rad 



+ Pl 



lost 



P, 



rad 



J*(r)J(r)dr 
E*(r) x H(r)dO 



Here, V is some volume that contains the ttansmitting 
antennas, J is the current density in the volume V 
and Q is some sufficiently smooth surface the interior 
of which contains V with dQ, denoting a surface area 
element. Also E and H are the electric and magnetic 
fields generated by the current density J and • x • denotes 
the vector product in IR 3 . 

Because of the vector product in the last equation 
above, the total power lost defines a norm on the space 
of square-integrable functions that does not admit an 
inner-product structure [18]. The theory developed in this 
paper is used to calculate the degrees of freedom of such 
spatial waveform channels in [18]. 

III. Main Results 

In this section we outline the main results of this paper. 
All the proofs of theorems are given in the Appendix. 

A. Degrees of Freedom for Compact Operators 

The definition of degrees of freedom at level e for 
compact operators on normed spaces is identical to the 
finite dimensional counterpart (Definition 12.11 ) discussed 
in the previous section with C n and C m replaced by 



general normed spaces. The following theorem ensures 
that the definition makes sense even in the infinite 
dimensional setting. 

Theorem 3.1: Suppose X and Y are normed spaces 
with norms and ||-||y, respectively, and T : X — > Y 
is a compact operator. Then for all e > there exisj^l 
N 6 IiQ and a set {4>i}f =l C Y such that for all x G 
Bi,x(0) 



inf 

ai ,...,ajy 



Tx 



< e. 



Y 

N 



Note that for N = the set {ipi}f =1 is empty and 
the sum in the above expression is void. We will use 
the following definition for the number of degrees of 
freedom at level e for compact operators on normed 
spaces. 

Definition 3.1 (Degrees of freedom at level e): 
Suppose X and Y are normed spaces with norms || • ||x 
and || • \\y, respectively, and T : X — > Y is a compact 
operator. Then the number of degrees of freedom of T 
at level e is the smallest N E Z,q such that there exists 
a set of vectors 
xeB ljX (0) 



{V>i, • • • , Vw} C Y such that for all 



inf 

a± ,...,ajv 



Tx 



< e. 



Y 



This definition has exactly the same interpretation as in 
the finite dimensional case: if there is some constraint 
|| • \\x < 1 on the space of source functions and if the 
receiver can only measure signals that satisfy || • ||y > e, 
then the number of degrees of freedom is the maximum 
number of linearly independent signals that the receiver 
can measure under these constraints. 

This definition however is a descriptive one and can 
not be used to calculate the number of degrees of free- 
dom for a given compact operator because the proof of 
Theorem 13 .H is not constructive. In the finite dimensional 
case we can calculate the degrees of freedom by calculat- 
ing the singular values. However, as far as we are aware, 
there is no known generalisation of singular values for 
compact operators on arbitrary normed spaces^. In the 
following subsection we will propose such a general- 
isation. In fact, we will use the degrees of freedom to 
generalise the concept of singular values. We will discuss 
the problem of computing degrees of freedom using 
generalised singular values in subsection IIII-DI below. 

5 Z, Zq and Z + are respectively the sets of integers, non-negative 
integers and positive integers. 

6 A generalisation to compact operators on Hilbert spaces is of 
course classical and well known. 
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Fig. 1. Degrees of Freedom of a Compact Operator 

Next, we establish some useful properties of degrees 
of freedom that will help motivate the definition of 
generalised singular values given in the next subsection. 

Theorem 3.2: Suppose X and Y are normed spaces 
with norms \\-\\x and || • ||y , respectively, and T : X — > Y 
is a compact operator. Let M(e) denote the number of 
degrees of freedom of T at level e. Then 

1) M(e) = for all e > ||T||. 

2) Unless T is identically zero, there exists an eo > 
such that Af(e) > 1 for all < e < e . 

3) M(e) is a non-increasing, upper semicontinuous 
function of e. 

4) In any finite interval (ei, e 2 ) CR, with < e\ < 
€2, A/"(e) has only finitely many discontinuities, i.e. 
M(e) only takes finitely many non-negative integer 
values in any finite e interval. 

The following two examples show that as e goes to 
zero, M(e) need not be finite nor go to infinity. 

Example 3.1: Let I 1 be the Banach space of all real- 
valued sequences with finite I 1 norm and let (e±, e 2 , ■ ■ ■) 
be the standard Schauder basis for I 1 . Define the operator 
T : I 1 — > I 1 by e n 1— > e± for all n G Z + . This operator is 
well-defined and compact and M(e) < 1 for all e > 0. 

Example 3.2: Let I 1 and (ei, e2, ■ ■ •) be defined as in 
the previous example. Define T : I 1 — > I 1 by e n 1— ► -e n 
for all n G Z + . Again T is well-defined and compact 
but lim^o A/"(e) = 00. 

Figure Q] shows a typical example of degrees of freedom 
at level e for some compact operator that satisfies all the 
properties in the above theorem. 

B. Generalised Singular Values 

We will identify the discontinuities in the number of 
degrees of freedom of T at level e with the (generalised) 
singular values of T. 



Definition 3.2 (Generalised Singular Values): 
Suppose X and Y are normed spaces and T : X — > Y 
is a compact operator. Let M(e) denote the number of 
degrees of freedom of T at level e. Then e m is the m th 
generalised singular value of T if 

sup e>(Em 7V*(e) = m — 1 and 
inf e<em M(e) = M > m. 

Further, if m < M then for all m < n < M, e n := e m 
is the n th generalised singular value of T. 
Note that by Theorem 13.21 part [3] we have 7V(e. m ) < 
m — 1 with equality if (but not only if) e m is not a 
repeated generalised singular value. 

Let the degrees of freedom of some operator T be as 
shown in Figure Q] Then the generalised singular values, 
e m , of T identify the jumps in the degrees of freedom. 
So, ei = 0.9, e 2 = e 3 = e 4 = 0.8, e 5 = 0.6, . . . 

Another way of understanding the connection between 
the number of degrees of freedom at level e and gener- 
alised singular values is as follows. 

Proposition 3.3: Suppose X and Y are normed 
spaces and T : X — > Y is a compact operator. Let M(e) 
denote the number of degrees of freedom of T at level e. 
Then Af(e) is equal to the number of generalised singular 
values that are greater than e. 

The intuition behind the definition for generalised sin- 
gular values needs further clarification. In the finite 
dimensional case, if a p is the p th singular value of some 
operator T : C n — > C m , then there exist corresponding 
left and right singular vectors v p £ C n and u p G C m 
such that v p is of unit norm, Tv p = u p and the norm 
of Up is a p . This is not necessarily true for arbitrary 
compact operators on normed spaces as the following 
example proves. 

Example 3.3: Let I 1 and (ei, e2, • ■ .) be defined as in 
Example 13.11 Define the operator T : l l — > I 1 by Te n = 
(1 — i)ei for all n G Z + . Then T is well-defined and 
compact. Also, the number of degrees of freedom of T 
at level e is 



if e > 1, 

1 if e < 1. 



So ei = 1. However, for any vector x in the unit sphere 

in I 1 , \\Tx\\ii < 1. 

The above example motivates the slightly more compli- 
cated statement in the following theorem which explains 
the intuition behind the definition of generalised singular 
values. 

Theorem 3.4: Suppose X and Y axe normed spaces 
with norms and respectively, and T : X — > Y 
is a compact operator. Let e m be a generalised singular 
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value of the operator T. Then for all 9 > there exists 
a <f> G X, \\4>\\x = 1, such that 



e m + 9> \\T4>\\ Y >e m -9. 
The above theorem shows how the generalised singular 
values are related to the traditionally accepted notion of 
singular values of compact operators on Hilbert spaces. 
In general, they are values the operator restricted to 
the unit sphere can get arbitrarily close to in norm. 
However, we still need to prove that in the special case of 
Hilbert spaces the new definition for generalised singular 
values agrees with the traditionally accepted definition 
for singular values. 

Recall that if Hi and H2 are Hilbert spaces with 
inner products (-, and (-, respectively and if 
T : Hi — > H2 is a compact operator then the Hilbert 
adjoint operator for T is defined as the unique operator 
T* :H 2 -> Hi that satisfies [20, Sec. 3.9] 



(Tx,y) 



H-2 



{x,T*y) Hl 



for all x £ Hi and y € H 2 - The singular values of T 
are defined to be the square roots of the eigenvalues of 
the operator T*T : Hi -> Hi. We will refer to these as 
Hilbert space singular values to distinguish them from 
generalised singular values. Note that we always count 
repeated eigenvalues or (generalised) singular values 
repeatedly. The following two theorems establish the 
connection between Hilbert space singular values and the 
number of degrees of freedom at level e. The theorems 
are important in their own right because they show that 
there are two other equivalent ways of calculating the 
degrees of freedom of a Hilbert space operator. 

Theorem 3.5: Suppose Hi and H2 are Hilbert spaces 
and T : Hi — > H2 is a compact operator. Then for all 
e > there exist an N € and a set of N mutually 
orthogonal vectors {4>i}f =l C Hi such that if 

x G Hi, H^llwi < 1 and (x,^)?^ = 



then 



\Tx\ 



< e. 



Moreover, the smallest N that satisfies the above condi- 
tion for a given e is equal to the number of Hilbert space 
singular values of T that are greater than e. 

Theorem 3.6: Suppose that Hi and H2 are Hilbert 
spaces and T : Hi — > H2 is a compact operator. Then 
the number of degrees of freedom at level e is equal to 
the number of Hilbert space singular values of T that 
are greater than e. 

As a corollary of Theorem 13.61 we get the following 
result. 

Corollary 3.1: Suppose Hi and H 2 are Hilbert spaces 
and T : Hi — > H 2 is a compact operator. Suppose {e m } 



are the generalised singular values of T and {a m } are 
the possibly repeated Hilbert space singular values of T 
written in non-increasing order. Then 

&m — Cm 

for all m € Z + . 

This corollary, reassuringly, proves that the generalised 
singular values are in fact generalisations of the tradi- 
tionally accepted notion of Hilbert space singular values. 
We will therefore use the terms generalised singular val- 
ues and singular values interchangeably unless specified 
otherwise for the remainder of this paper. 

In Hilbert spaces we have three characterizations for 
degrees of freedom: 1) as in Definition 13.21 2) as in 
Theorem 13.61 in terms of singular values and 3) as in 
Theorem 13.51 in terms of mutually orthogonal functions 
in the domain. 

We have used the first two characterisations in the 
generalisation to normed spaces. However, the final char- 
acterisation is more difficult to generalise. It would be 
extremely useful to generalise the final characterisation 
because, for the Hilbert space case, the functions 4>i in 
Theorem 13.51 are in some sense the best functions to 
transmit (see e.g. [14]). One could possibly replace the 
mutual orthogonality by almost orthogonality using the 
Riesz lemma (see e.g. [20, pp. 78]). 

Lemma 3.7 (Riesz's lemma): Let Y and Z be sub- 
spaces of a normed space X and suppose that Y is closed 
and is a proper subspace of Z. Then for all 9 £ (0, 1) 
there exists a z 6 Z, ||z|| = 1, such that for all y € Y 

\\y-A\ ^ e - 

The following conjecture is still an open question. 

Conjecture 3.1: Let X and Y be reflexive Banach 
spaces and let T : X — > Y be compact. Given any e > 
and some 9 € (0, 1), there exists a finite set of vectors 
{(t>i}f=i C X such that for all x € X, \\x\\x < L 



inf 



N 

£■ 

i=l 



> 



(1) 



X 



implies 

ll^lly < e- 

Comparing with Theorem l3.5[ condition ([J) is analogous 
to requiring that x be orthogonal to all the <pi. The con- 
jecture is definitely not true unless we impose additional 
conditions such as reflexivity on X and/or Y as the next 
example proves. 

Example 3.4: Let I 1 , (ei,e2,...) and the compact 
operator T : I 1 — > I 1 be defined as in Example 13.11 
Now let e < 1. For any x = ^2 n a n e n € i 1 , if ||x|| = 1 
and if a n > for all n then ||Tx|| = llxll = 1 > e. 
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Hence no finite set of vectors can satisfy the conditions 
in the conjecture. 

In the following subsection, we use degrees of freedom 
and generalised singular values to define the essential 
dimension of a communication channel. 

C. Essential Dimension for Compact Operators 

The definition for degrees of freedom given in Sec- 
tion JIFA] depends on the arbitrarily chosen number e and 
therefore this definition does not give a unique number 
for a given channel. The physical intuition behind choos- 
ing this arbitrary small number e is nicely explained in 
Xu and Janaswamy [12]. In that paper e = a 2 denotes the 
noise level at the receiver and the authors state that the 
number of degrees of freedom fundamentally depends 
on this noise level. 

However, in several important cases the number of de- 
grees of freedom of a channel is essentially independent 
of this arbitrarily chosen positive number [4,9,11,13,14, 
16]. This is due to the fact that in these cases the singular 
values of the channel operator show a step like behavior. 
Therefore, for a big range of values of e, the number 
of degrees of freedom at level e is constant. This leads 
us to the concept of essential dimensionality^ which is 
only a function of the channel and not the arbitrarily 
chosen positive number e. Some of the properties that 
one might require from the essential dimension of a 
channel operator are: 

1) It must be uniquely defined for a given operator 
T. 

2) The definition must be applicable to a general class 
of operators under consideration so that compar- 
isons can be made between different operators^] 

3) It must in some sense represent the number of 
degrees of freedom at level e. 

The last requirement above needs further clarification. 
Obviously the essential dimension of T can not in 
general be equal to the number of degrees of freedom at 
level e because the latter is a function of e. However, if 
the singular values of T plotted in non-increasing order 
change suddenly from being large to being small then the 
number of degrees of freedom at the "knee" in this graph 
is the essential dimension of T. The following definition 
for the essential dimension tries to identify this "knee" 
in the set of generalised singular values. 

7 Note that the term "essential dimension" has been used instead 
of "degrees of freedom" in several papers. As far as we are aware, 
this is the first time an explicit distinction is being made between the 
two terms. 

8 This requirement is in contrast to the essential dimension defini- 
tion in [17] that is only applicable to the time-bandwidth problem. 



Each level e defines a unique number of degrees 
of freedom Af(e) for a given compact operator T. So 
for each positive integer n 6 Z + we can calculate 
E(n) = fi({e : n = M(e)}). Here /i(-) is the Lebesgue 
measure. The function E(n) is well defined because of 
the properties of generalised singular values discussed in 
Theorem l3.2l We can now define the essential dimension 
of T as follows. 

Definition 3.3: The essential dimension of a compact 
operator T is 

EssDim(T) = argmax{£'(n) : n € Z + } 

where E{n) is defined as above. If argmax above is 
not unique then choose the smallest n of all the n that 
maximise E(n) as the essential dimension. 
In this definition we are simply calculating the maximum 
range of values of the arbitrarily chosen e over which the 
number of degrees of freedom of an operator does not 
change. It uniquely determines the essential dimension of 
all compact operators. Further, it is equal to the number 
of degrees of freedom at level e for the maximum range 
of e. Choosing this value for the number of degrees 
of freedom in order to model communication systems 
has the big advantage that it is independent of the noise 
level at the receiver. Further, if for a given noise level 
the number of degrees of freedom is greater than the 
essential dimension then one can be sure that even if the 
noise level varies by a significant amount the number 
of degrees of freedom will always be greater than the 
essential dimension. 

The essential dimension of T is the smallest number of 
generalised singular values of T after which the change 
in two consecutive singular values is a maximum. One 
could also look at how the generalised singular values are 
changing gradually and the above definition is a special 
case of the following notion of essential dimension of 
order n, namely the case where n = 1. 

Definition 3.4: Let X, Y be normed spaces and let 
T : X — > Y be a compact operator. Let {e m } be the set 
of generalised singular values of T numbered in non- 
increasing order. Then define the essential dimension of 
T of order n to be iV if n is even and 

e N-n/2 - e N+n/2 > e M-n/2 ~ e M+n/2 

for all M 7^ N. If there are several N that satisfy the 
above condition then choose the smallest such TV. If n 
is odd then choose the smallest N that satisfies 

e 7V-(n-l)/2 — e AT+(n+l)/2 > e M~(n-l)/2 ~ e A-/+(n+l)/2 

for all M ^ N. 

A simple example illustrates the concepts of essential 
dimensionality and degrees of freedom. 
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Fig. 2. Singular values of an Operator 



Example 3.5: Figure [2] shows the singular values of 
some operator T. For this operator the number of degrees 
of freedom at level 0.75 is 7 and at level 0.1 is 8. 

The essential dimension of the channel is 7. This 
is because for e G [0.4,0.8), 7V(e) = 7. Therefore 
E(7) = 0.4 which is greater than E(n) for all The 
essential dimension of order 2 is 8 because ej — eg = 0.7 
which is greater than €m-i — eju+i for all M ^ 8. 

D. Computing generalised singular values 

Both, degrees of freedom and essential dimension 
for a communication channel, can be evaluated if the 
generalised singular values of the operator T describing 
the channel are known. However, no known method 
exists for computing these singular values for general 
compact operators. In this section, we develop a numer- 
ical method, based on finite dimensional approximations, 
that could be used to calculate generalised singular 
values. 

Theorem 3.8: Suppose X and Y are normed spaces 
and T : X — > Y is a compact operator. Also suppose 
that X has a complete Schauder basis {cfti, ^2, . . .} and 
let S n = span{0i, . . . , cj) n }. Let T n = T\ Sn : S n -> Y, 
n G Z + . If e m , the m th singular value of T, exists then 
for n large enough e m n , the vnP 1 singular value of T n , 
will exist and 



lim e 

n— >oo 



m,n 



If e m , n exists then it is a lower bound for e m . 

The theorem shows that if the domain of the operator 
has some complete Schauder basis then we can calculate 
the generalised singular values of the operator restricted 
to finite dimensional subspaces and as the subspaces 
get bigger we will approach the singular values of the 
original operator. Moreover, the theorem also proves that 



the singular values of the finite dimensional operators 
provide lower bounds for the original generalised singu- 
lar values. We, however, still need a practical method of 
calculating the singular values of linear operators defined 
on finite dimensional normed spaces. 

Let X, Y be two finite dimensional Banach spaces and 
let T : X — ► Y be a linear operator. Suppose ei, . . . , e n 
are the generalised singular values of T and denote B\ = 
{x e X : \\x\\x < 1}- We know that for all e > e p+ i, 
AZ'(e) < p. Hence for each e > e p+ \ there exists a set 
{V>i}f=i CY such that 



sup inf 



Tx 



i=l 



< e. 



Y 



Let ty p e denote the set of all sets {fy : 



Y that satisfy the above inequality for a given e > e p _| 
and let 



U 



p,e- 



e>e„ 



With this notation we can now prove that the generalised 
singular values of a linear operator defined on a finite 
dimensional normed space can be expressed as the 
solution of an optimisation problem. 

Theorem 3. 9: Let X, Y be two finite dimensional 
Banach spaces and let T : X — > Y be a linear operator. 
Also let B\ be the closed unit ball in X and suppose ^ p 
is defined as explained above. Then 



sup ||Ta;||y 



and for all p G 



inf sup inf 



Tx 



e p+l- 



Given the "correct" set of functions ipi, the above 
theorem characterises the singular values in terms of a 
maximisation problem over a finite dimensional domain. 
It is however difficult to check whether a given set of 
functions {i/^i}^ =1 is an element of We therefore 
propose the following algorithm to calculate bounds on 
the generalised singular values. 

Suppose X,Y, T : X — > Y, ei, . . . , e n and B\ are 
defined as in Theorem 13.91 Let 

e'i = sup ||Tx||y. 

Because B\ C X is a compact set and || • \\y and T are 
continuous, there exists an x\ G B\ such that ||Txi ||y = 
e^. Choose ipi = Tx\. 

Now suppose ipi , . . . , ip p have been chosen. Then let 

p 

Tx-^ a itpi ■ ( 2 ) 



e p+ i = sup inf 



i=l 



Y 



10 



Again, because B\ C X is a compact set and || • \\y 
and T are continuous, there exists an x p+ \ G B\ such 
that x p+ \ attains the maximum in the above equation. 
Choose Vv+i = Txp+i. Comparing with Theorem 13.91 
we note that e' p+1 is an upper bound for e p+ \. It is an 
open question as to whether e' p+1 = e p +i. 

In this algorithm, instead of searching over all possible 
sets in ^ p we select a special set that is in some sense 
(it consists of images of the x G B\ that attain the 
maximum in equation ©) the best possible set to use. 
This choice is essential because otherwise the calculation 
of generalised singular values becomes too cumbersome 
(one needs to find the set ^ p before calculating e p+ \). 
Note however, that the above algorithm gives the correct 
value for e\. 

The theory presented here has been used to compute 
the generalised singular values and degrees of freedom 
in spatial waveform channels of the type discussed in 
Example 12.31 The results of these computations are 
presented in Somaraju [18]. Due to space constraints, 
these results are not further discussed in this paper. 

E. N on- compactness of channel operators 

Throughout this paper we have exclusively dealt with 
channels that can be modeled using compact operators. 
We have done so because of the following result. 

Theorem 3.10: (Converse to Theorem 13.11 ) Suppose 
X and Y are normed spaces with norms || • \\x and 
|| • \\y, respectively, and T : X — > Y is a bounded linear 
operator. If for all e > there exist N € Zq" and a set 
{tpi}f =1 C Y such that for all x G B ljX (0) 



inf 



N 



Tx 



< e 



Y 



then T is compact. 

So any bounded channel operator with finitely many 
sub-channels must be compact. Indeed, if one can find 
a channel that is not described by a compact operator, 
then it will have infinitely many sub-channels and will 
therefore have infinite capacity. Also, if the channel is 
described by an operator that is linear but unbounded 
then there will obviously exist sub-channels over which 
arbitrarily large gains can be obtained]^ 

IV. Conclusion 

In this paper we assume that a communication channel 
can be modeled by a normed space X of transmitter 

9 It could hence be argued that non-compact channel operators are 
unphysical, however, we will leave it to the reader to make this 
judgement. 



functions that a transmitter can generate, a normed space 

Y of functions that a receiver can measure and an 
operator T : X — > Y that maps the transmitter functions 
to functions measured by the receiver. We then introduce 
the concepts of degrees of freedom at level e, essential 
dimension and generalised singular values of such chan- 
nel operators in the case where they are compact. One 
can give a physical interpretation for degrees of freedom 
as follows: if there is some constraint || • \\x < 1 on 
the space of source functions and if the receiver can 
only measure signals that satisfy || • ||y > e then the 
number of degrees of freedom is the number of linearly 
independent signals that the receiver can measure under 
the given constraints. If the degrees of freedom are 
largely independent of the level e then it makes sense to 
talk about the essential dimension of the channel. The 
essential dimension of the channel is the smallest number 
of degrees of freedom of the channel that is the same for 
the largest range of levels e. We show how one can use 
the number of degrees of freedom at level e to generalise 
the Hilbert space concept of singular values to arbitrary 
normed spaces. We also provide a simple algorithm that 
can be used to approximately calculate these generalised 
singular values. Finally, we prove that if the operator 
describing the channel is not compact then it must either 
have infinite gain or have an infinite number of degrees 
of freedom. The general theory developed in this paper is 
applied to spatial waveform channels in Somaraju [18]. 

Appendix 

Proofs of Theorems: Theorem \3.l\ Suppose X and 

Y are normed spaces with norms || • \\x and || • ||y, 
respectively, and T : X — > Y is a compact operator. 
Then for all e > there exist N G 1^ and a set 
{A}f=i C Y such that for all x G Bi,x(0) 



inf 

ai,...,djv 



N 



i=i 



< e. 



(3) 



Proof: The proof is by contradiction. Let e > be 
given. Suppose no such N exists. 

Let x\ G Bi^x(fy be any vector. Choose ip\ = Tx\. 
Suppose that {x\, . . . ,xx} and {ipi, . . . , i/j^} have been 
chosen. Then, by our assumption, there exists an x^+i G 
Bi,x(ty such that 



inf 



N 



Tx N+ i - ^2 aiipi 



i=l 



> e. 



(4) 



r 



Choose ipN+i = Txn+i- By induction, for M < N we 
have 

\\Tx N+ i - Tx m \\y > e. 



1 1 



This follows from © by setting cjj = 0, i < N, i ^ M, 
and ajvf = 1. Therefore, using the Cauchy criterion, 
the sequence {Tx n }^ =1 chosen by induction cannot 
have a convergent subsequence. This is the required 
contradiction because {i„}™ =1 is a bounded sequence 
and T is compact. ■ 
Theorem 13.21 Suppose X and Y are normed spaces 
with norms \\-\\x an d \\-\\y, respectively, and T : X —> Y 
is a compact operator. Let J\f(e) denote the number of 
degrees of freedom of T at level e. Then 

1) M(e) = for all e > ||T||. 

2) Unless T is identically zero, there exists an eo > 
such that J\f(e) > 1 for all < e < e . 

3) M(e) is a non-increasing, upper semicontinuous 
function of e. 

4) In any finite interval (€1,62) C R, with < e\ < 
£2, A/"(e) has only finitely many discontinuities, i.e. 
M{e) only takes finitely many non-negative integer 
values in any finite e interval. 

Proof: 

1) Because T is compact it is bounded, and therefore 
||T|| < 00. Suppose e > ||T|| then \\Tx\\ Y < 
\\T\\ < e for all x G B ljX {0). Therefore 7V(e) = 0. 

2) If ||T|| > there exists an x G X, \\x\\x < 1 such 
that \\Tx\\ Y > 0. Set e := \\Tx\\ Y . Then for all 
< e < e , M(e) > 1. 

3) Suppose < ei < 62. Then there exist functions 
ipi, ... , V'jV(ei) sucn that for all x G 5x^(0) 



inf 

ll-l%(q) 



Tx 



< ei < £2 



Therefore N{t2) < -^( e i) from the definition of 
the number of degrees of freedom at level e, i.e. 
M(e) is non-increasing. In particular we have 

lim M{e) <M(ex). 
e\ei 

Assume that the above inequality is strict. Then 
there exists an N G Zj, N < N(e±), and for all 
9 > there exists a set lili C Y such that 
for all x G Si )X (0) 



inf 

cli ,...,ajv 



A 1 



< ei + 6. (5) 



Y 



On the other hand, since > N, for all sets 

{ij)i\f =1 C V there exists an x G i?i.x(0) such 
that 



// := inf 
aij...,djv 



Tx - ^ Oj^j 



> ei. 



(6) 



But (f5]) contradicts © for := — ei). Hence 
lim e \ £l jV(e) = A/"(ei) and AA(e) is upper semi- 
continuous. 
4) This follows from Parts Q] and [3] 



Proposition \3.3\ Suppose X and Y are normed spaces 
and T : X — > Y is a compact operator. Let AZ'(e) denote 
the number of degrees of freedom of T at level e. Then 
M{e) is equal to the number of generalised singular 
values that are greater than e. 

Proof: This follows from careful counting of the 
numbers of degrees of freedom at level e including re- 
peated counting according to the height of any occurring 
"jumps". ■ 

Theorem \3.4\ Suppose X and Y are normed spaces 
with norms an d \Y\\y, respectively, and T : X —> Y 
is a compact operator. Let e m be a generalised singular 
value of the operator T. Then for all 6 > there exists 
a cf) G X, || 1| jsc = 1, such that 



e m + 6> 



Y > e r 



Proof: The proof is by contradiction. Assume that 
there exists a 6 > such that for all <f> G X, \\(p\\x = L 
we have \\T4>\\ Y [e m -0,e m + 9]. Let Af(e) denote the 
number of degrees of freedom at level e of the operator 
T. From the definition of degrees of freedom at level e 
we have 



M{e m + 9) < 
M{e m -9) > 



in 
111. 



1. 



(V) 
(8) 



By ©, there exist vectors ipi, . . . , Vv«-i G Y such that 
for all x G Bi,x(0) 



inf 

ai,...,a m _i 



Tx 



m—l 

E 

i=i 



< e m + 



By our assumption on 



inf 

ai ,...,a m 



Y- 



m—l 



< e r 



Y 



This follows from consideration of the case a± = ■■■ = 
a m -i = 0. Hence M(e rn — 9) < m — 1 since scaling (f> 
to non-unit norm is equivalent to scaling all the aj. This 
contradicts inequality ([8]). Therefore there exists a that 
satisfies the conditions of the theorem. ■ 
Theorem \3.5\ Suppose TL\ and TL2 are Hilbert spaces 
and T : H\ — > H2 is a compact operator. Then for all 
e > there exist aniVe Z^ and a set of N mutually 
orthogonal vectors {(fti}f = i C Hi such that if 

x G Hi, ||x||ftj < 1 and (x, (f>i)Hi = 
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then 



I Tail 



< e. 



Moreover, the smallest N that satisfies the above condi- 
tion for a given e is equal to the number of Hilbert space 
singular values of T that are greater than e. 

Proof: We first prove that such an N is given by 
the number of Hilbert space singular values of T that 
are greater than e and then prove that this is the smallest 
such N. 

Let e > be given. Because T is compact, we can 
use the singular value decomposition theorem which says 
[21, p. 261] 



T- = ^2a i (-,4> i ) Hl ^ i . 



(9) 



Here, <jj, fa and ipi with i G Z + are the Hilbert space 
singular values and left and right singular vectors of T, 
respectively. We assume w.l.o.g. that the Hilbert space 
singular values are ordered in non-increasing order. We 
denote by N\ G Z + the number of Hilbert space singular 
values of T that are greater than e, i.e. a\ > e if and only 
if i < Nl 

Now, if x is orthogonal to fa,i = 1,...,N\ and if 
IMIwi < 1 then from equation (O 

oo 

\\ TX \\H2 = E^K^^WiPll^illWa 
i=l 

oo 
i=JVi+l 

< e 2 - 

For N < N\, the linear span of any set {(pi}fL 1 C Hi 
has a non-trivial orthogonal complement in the span of 
{4>i\f=i- Any vector x in this complement with HxH?^ = 
1 fullfills the conditions of the theorem but ||Tx||-^ 2 > e 
by equation ©. ■ 

Theorem 13.61 Suppose that H\ and H2 are Hilbert 
spaces and T : Hi — ► W2 is a compact operator. Then 
the number of degrees of freedom at level e is equal to 
the number of Hilbert space singular values of T that 
are greater than e. 

Proof: As in the prove of the previous theorem, let 
N\ G 1> + denote the number of Hilbert space singular 
values of T that are greater than e. Let <jj, fa and tpi with 
i £ Z + denote the Hilbert space singular values in non- 
increasing order and the left and right singular vectors 
of T, respectively. Let N2 G Z + denote the number of 
degrees of freedom of T at level e. 

We first prove that N\ > N2.lt x is in the unit ball in 
Hi then we can write x = ^i=i( x 5 fa)n.ifa + x r- Here 



x r is the remainder term that is orthogonal to all the fa. 
From equation (© and <7j < e for i > N\ it follows that 



Tx - ^2ai{x,fa) ni il)i 
i=i 



< e 



and hence 7V*i > N2 by the definition of the number of 
degrees of freedom at level e (set aj = ^{xjfa)^ in 
that definition). 

To prove that N\ < N2 assume that 7V*i > N2 to arrive 
at a contradiction. Then there exists a set {tp^}^ C H2 
such that 



inf 



N 2 
i=l 



< e 



W 2 



for all x G Hi, < 1- Because we assume that 

Ni > N2, there exists a y € span{V>i, ■ ■ ■ , ^jvi} which 
is orthogonal to all the i/A. Let y = Si=i b%^%. Then 
y = Tx where x = X^i=i by equation I©. We 
can assume w.l.o.g. that the 6j are normalised so that 
Haijl^j = 1. If this is done then 



inf 



A f 2 



Tx - ^ OiVi 



i=l 



H 2 



> 



\y\\n 2 



£* 

i=l 

e 2 . 



(10) 

(11) 

(12) 
(13) 



In the above we get equation (TTOb from the fact that y 
is orthogonal to all the ip^, inequality (fT2l from <jj > e 
for i < N\ and equation (fT3l from ||a;||fti = 1. The 
inequality (fT0l)-(fT3l is the required contradiction. This 
proves that N\ < N2 and hence N± = N2. ■ 
Corollarv \3.1\ Suppose H\ and H2 are Hilbert spaces 
and T : H\ — > H2 is a compact operator. Suppose {e m } 
are the generalised singular values of T and {a m } are 
the possibly repeated Hilbert space singular values of T 
written in non-increasing order. Then 



for all m G 1> + . 

Proof: This follows immediately from Theorem |3.6| 
and Proposition 13.31 by a simple counting argument. ■ 
Theorem \3.8\ Suppose X and Y are normed spaces 
and T : X — > Y is a compact operator. Also suppose 
that X has a complete Schauder basis {fa, fa, . . .} and 
let S n = sp&n{fa, ... , 4> n }. Let T n = T\ Sn : S n -> F, 
n G Z + . If e m , the m </l singular value of T, exists then 
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for n large enough e mi „, the m th singular value of T n , 
will exist and 

lim € m n — 6 m . 

n— >oo 

If e m , n exists then it is a lower bound for e m . 
Proof Outline: The crux of the argument used to prove 
the theorem is as follows. Assume e > is given and 
let M{e) denote the number of degrees of freedom at 
level e for the operator T. By definition there exist 
functions {ipi, . . . , 4>Af(t)} c ^ such that for all x G X, 
\\x\\x < 1> rx can be approximated to level e by 
a linear combination of the ipi and further, no set of 
functions {ip[, . . . , i/^-} C Y can approximate all the Tx 
if iV < 7V*(e). Equivalently, there is a vector in the closed 
unit ball in X whose image under T can be approximated 
by a vector in spanj^i , . . . , VW(e) } but not by any vector 
in span^i,...,^}. 

So we take the inverse image of an e-net of points in 
spanj^i, . . . , ipj\f(e)} an d choose n large enough so that 
all the inverse images are close to S n . We can do this 
because the fa form a complete Schauder basis for X. 
We then show that there exists a vector in S n such that 
its image under T cannot be approximated by a linear 
combination of . . . , tp' N for N < 7V(e). This will 
prove that the number of degrees of freedom at level e 
of T n approaches that of T and consequently so do the 
singular values. The details are as follows. 

Proof: We will prove this theorem in two parts. 
Assume that e m exists. In part a) we will prove that if 
e m ,N exists for some N G Z + then e m ^ n exists for all 
n > N, and the e m ^ n form a non-decreasing sequence 
indexed by n that is bounded from above by e m . In part 
b) we prove by contradiction that e m>n exists for some 
n G Z + and that e m n must converge to e m . 

We will use the following notation in the proof: 



Therefore for all e > 



spangj'i/'i 



• ■ ,V>iv} = 
{y £Y: 



inf 

a\ ,...,ajv 



y 



N 

1=1 



Y 



and B r = {x G X : < r}. 

Part a: Let T and T n be defined as in the theorem 
and let 7V(e) and N n (e) be the numbers of degrees of 
freedom at level e of T and T n , respectively. Assume 
that e m ,n^ exists and let ti2 > n\. 

Then for all sets {ipi, . . . , ip_\f n ( e )-i} C y there is a 
£ G <S ni H -Bi such that 

r ni C = T££ span e {V>i,. • . , (e)-l}- 

Because 5 ni C <S' n2 we have £ G 5„ 2 n B\ and 

7"n 2 £ = Ti i span £ {^i,. . .,^jv ni ( e )-i}. 



M n2 (e) >M ni {e). 



Because 



inf Af ni (e) > m 



(14) 



(15) 



we have N n2 (e) > N ni (e) > m for e < e m>ni . Hence 
em,n 2 must exist. 

From the definition of generalised singular values we 
have inequality (fT3T ) and 



sup 



M n2 (e) <m-l 



If em,n! > £tn,n 2 then there exists an e' such that e m>ni > 
e' > e m n 2 - Therefore, 



A/" ni (e') > m > m 



l>M n2 {e' 



This contradicts inequality (TT4b . Therefore e mni < 

The same line of arguments as above can be used to 
show that if both e. m and e mj „ exist then e mj „ < e m . 
Recall that we have assumed at the beginning that e m 
exists. Therefore, if e m) N exists for some N G Z + then 
e m ^ n is a non-decreasing sequence in n > N that is 
bounded from above by e m . 

Part b: By part a), if e mj „ exists for n > n\ then, 
because e m . n is a bounded monotonic sequence in n it 
must converge to some e' m < e m . 

Now there are two situations to consider. Firstly, e m>n 
might not exist for any n G Z + . Secondly, e m ,n might 
exist for some n but the limit e' m might be strictly less 
than e m . We consider the two situations separately and 
arrive at the same set of inequalities in both situations. 
We then derive a contradiction from that set. 

Situation 1: Assume that e m n does not exist for any 
n G Z + . Then 

M n {e)<m-l (16) 

for all n G Z + and e > 0. Using the definition of degrees 
of freedom for T there exist constants a < (3 < e m such 
that 



A/" n (a) < m - 
N((3) > m. 



1 for all n G 7L + and 



(17) 
(18) 



Situation 2: Assume that e' < e m . From the defini- 



tion of generalised singular values we know 

su Pe>e m „ < m — 1 for all n G \ 

inf e<em M(e) > m. 



and 
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Because 

£m n £rn> know that there exist numbers 
a and 8, e' m < a < 8 < e m such that 

M n (a) <m-l for all n G Z+ and (19) 
N(B) > m. (20) 

These are the same conditions as (TTTT ) and (TT8T ). There- 
fore, in both situations we need to prove that the inequal- 
ities (fT9l and (1201 cannot be simultaneously true. 

Because T is compact, TB\ is totally bounded [20, 
ch. 8]. Therefore, TB\ has a finite e-net for all e > 0. 
Hence there exists a set of vectors . . . ,£p} C B\ 
such that for all y G TBi there exists a p, 1 < p < P 
with 

l|re P -I/||y <^f L - (2D 

Now, because {0i, 02, . . .} is a complete Schauder basis 
for X and because P < oo, there exists a number N 
such that for all n > N and for all p, 1 < p < P, there 
exists a £ Pin € 5 n fl B\ such that 

/3 — a 



\\€p,n - €p\\x < 



2\\T\ 



(22) 



Therefore, for all y G TB\ and for all n > N there 
exists a p, 1 < p < P and a £ Pjn G S n n B\ such that 



y||y = \\n P , n - T£ p + Ti P - y\\ Y 

< \\T£ Py n - T£ p \\y + \\T£ P - y\\ Y 

< \m P , n - Q\\Y + ^ 



. , 8-a B-a 
11 11 2||T|| 2 
= 8 — a. 



(23) 



We get the first inequality above from the triangle 
inequality, the second one from inequality (|2TT > and the 
final one from inequality (l22l . From inequality ( fT9l and 
the definition of the number of degrees of freedom, we 
know that for all n G Z + there exists a set of vectors 
{V>i,n, • • • , 0m-l,n} C Y such that 



y G span a {V>i,n, • • • , ^m-l,n} 



(24) 



for all y G T(5„ n5i). 

But, from the definition of the number of degrees of 
freedom and inequality (1201 we know that for all n G Z + 
and all sets of vectors {ipi jn , ■ ■ ■ , ipm-i,n} there exists a 
vector ip G TB\ such that 

■0 span /3 {^i in , . . . , "i/Vn-l.n}- 



From inequality (1231 we know that for all n > N there 
exists a £ Pi „ G S n fl i?i such that 

\\T£p, n - ip\\ < B - a. 



Therefore, for all n > N there exists a £ Pi „ G 5 n fl B\ 
such that 

^p,n ^ Span Q {V>l,n, • • • , 1pm-l,n}- (25) 



This directly contradicts condition (1241 . Therefore, if e m 
exists then e m)n exists for n large enough and 



lim e 



m.n — ^m- 



Theorem 15.91 Let X, Y be two finite dimensional 
Banach spaces and let T : X — > Y" be a linear operator. 
Also let i?i be the closed unit ball in X and suppose ^ p 
is defined as in Section IIII-Di Then 



sup \\Tx\\ Y = ei 



and for all p G 



inf sup inf 



Tx - ^ Qilpi 



i=l 



Proof: Let e p+1 denote the left hand side of the 
above equation. Assume e' p+1 < e p+ \. Then there exists 
a set {ijjiY i=l G ^ p such that 



€p + l : = sup inf 
xeBi oi,...,o p 



i=l 



< e 



p+i- 



By definition this implies 7V(e p+1 ) < p, a contradiction 
to inf e<ep+1 jV(e) > p + 1. Hence e p+1 > e p+1 . Now 
assume e p+1 > e p +i. Let e G (e p+ i,e p+1 ). From 
su Pe>e p+ i -^( e ) — P it follows JV(e) < p. Hence there 
exists a set {^i}f =1 C Y" such that 



sup inf 



Tx - ^2 (nipt 



i=l 



< e < ei 



P+i- 



Therefore {^i}f =1 G C and 



e' + i > inf sup inf 

W.}Lie*pxeB 1 a i-^ 



i=l 



a contradiction. Hence e p+1 = e p+ i. ■ 
Theorem \3.10\ (Converse to Theorem 13.11 ) Suppose X 
and Y" are normed spaces with norms || • \\x and || • 
respectively, and T : X — > Y is a bounded linear 
operator. If for all e > there exist N G Zq" and a 
set {V'ilili C Y such that for all x G i?i,x(0) 



inf 



JV 



i=i 



< e 



r 



then T is compact. 

Proof: We prove that T is compact by showing 
that the set T(B 1:X (0)) is totally bounded. Let 5 > be 
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given. Then there exist an N G Zq" and a set {ipi}^ C 
F such that for all x G B 1>x {0) 



inf 

a 1 ,...,a N 



A 1 



Tx — a i^i 



i=l 



Y 



< 5 -. 
4 



(26) 



For any given x G £?i 5 x(0) we can choose af, i 
1, 



, iV such that 

Tx-Y,<A 

i=l 

Tx - ^2 ciiipi 



N 



Y 



< inf 

ai ,...,ajy 



N 



i=l 



r 



5 

+ 4 



< 



(27) 



(28) 



Here, the last inequality follows from (126I ). Also, because 
we can choose a* = for i = 1,...,N, for all x G 

^i,x(0) 



inf 

ai,...,ajv 



/V 



i=l 



< \\Tx\\ Y . (29) 



Substituting inequality (1291 into (|27T ) and using the 
triangle inequality, we get 



/V 



i=l 



i, , ^ „ ,, S 

< 2\\Tx\\ Y + - < 2\\T\\ + -. (30) 



We get the last inequality from the boundedness of T. 
Because the span of ipi,...,ipN is finite dimensional 
and because of the uniform bound (l30l) . there exists a 
finite set of elements {y\, . . . ,um} C Y such that for 
all x G B~i,x(0) 



inf 

i=l,...,M 



5 

< -. 

~ 2 



(31) 



N 
3=1 

From inequalities ( f3TT > and d28l ) and the triangle inequal 
ity we get for all x G B\ : x(0) 



Y 



inf — Txlly < 5. 
i=l,...,M 



(32) 



Therefore, the j/j, i = 1,...,M form a finite <5-net 
for T(B ljX (0)) and therefore T(5i jX (0)) is totally 
bounded. Hence, T is compact. ■ 
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